Composing Pick-and-Place Tasks by Grounding Language

Controlling robots to perform tasks via natural language is one of the most challenging topics in human-robot interaction. In this work, we present a robot system that follows unconstrained instructions pick and place arbitrary objects effectively resolves ambiguities through dialogues. Our approach infers their relationships from input images expressions can accordance with spatial relations expressed by user. Unlike previous approaches, consider grounding not only for picking but also placement everyday language. Specifically, relations, allow specification complex instructions, e.g. “place it behind middle red bowl”. results obtained using real-world PR2 demonstrate effectiveness our method understanding pick-and-place sequentially composing them solve tabletop manipulation tasks. Videos are available at http://speechrobot.cs.uni-freiburg.de .